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Nearly 20 years into the era of results-based accountabil¬ 
ity, a new generation of afterschool accountability systems 
is emerging. Rather than aiming to test whether programs 
have produced desired youth outcomes, an increasing 
number of afterschool funders and sponsors are shaping 
more flexible, collaborative, and lower-stakes accountabil¬ 
ity systems. 

Could they do even more? By designing accountabil¬ 
ity systems that fully embrace the notion of afterschool 
programs as learning organizations and by using research 
from organizational development, education, and youth 
development to create effective learning environments, 
funders and sponsors can help programs to improve 
quality—and therefore, to succeed in their goal of 
achieving better outcomes for young people. 


Accountability in the Age of Outcomes 

The 1990s, a time of national investment in afterschool, 
were also a time of increasing accountability. Passage of 
the Government Performance and Results Act of 1993 
ushered in a new era of results-based accountability (Of¬ 
fice of Management and Budget, 1993). Programs could 
no longer count delivery of services as evidence of dol¬ 
lars well spent; funders expected to see measurable 
youth outcomes (Fuhrman, 1999; Kane, 2004; Walker 
& Grossman, 1999). These outcomes were driven not 
by the goals of afterschool programs but by the inter¬ 
ests of constituent groups that were looking to after¬ 
school to solve societal ills ranging from poor academic 
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performance to juvenile delinquency (Halpern, 2005). 
To ensure the value of their investments, many funders 
created accountability systems to test whether programs 
were producing the desired youth outcomes. 

Eager for new private and public dollars, many af¬ 
terschool programs began collecting data on youths 
standardized test scores, grades, school attendance, and 
delinquency records, even when these outcomes didn’t 
align with what programs were trying to accomplish. As 
large cities expanded the numbers of afterschool slots 
and sites, compliance led to creation of large-scale data 
management systems, new technologies such as swipe 
cards, and new mechanisms for gaining access to pub¬ 
lic school records. Some afterschool programs created 
administrative positions dedicated 
solely to managing youth outcome 
data (Fiester, 2004). 

Just a few years into the out¬ 
comes accountability era, warning 
signals emerged. Researchers ques¬ 
tioned whether the outcomes cho¬ 
sen by funders were appropriate, 
realistic, or even desirable goals 
for afterschool programs. Some 
expressed concern that funders 
hadn’t acknowledged the supports 
programs would need to yield the 
results funders were looking for 
(Walker & Grossman, 1999). Oth¬ 
ers argued that outcomes measure¬ 
ment offered an opportunity for 
programs to focus on learning and 
improvement toward outcomes 
that were meaningful to their mis¬ 
sion (Schilder, Horsch, Little, Bra¬ 
dy, & Riel, 1998; Surr, 2000). Despite these reactions, 
youth outcomes—with an increasingly academic focus— 
remained a key focus for many accountability systems. 
When funding for 21st Century Community Learning 
Centers (21st CCLC) was included in the No Child Left 
Behind Act of 2001, afterschool programs became even 
more explicitly focused on helping at-risk youth achieve 
school success. 

In his paper Confronting the Big Lie (2005), Robert 
Halpern railed against the trend toward an academic fo¬ 
cus in afterschool programs. Citing the 2004 evaluation 
of 21st CCLC (Dynarski et al., 2004), which failed to 
show academic effects for participating youth, Halpern 
argued that schools, not afterschool programs, should be 
accountable for academic outcomes. Forcing programs 


to focus on academic outcomes sidetracked them from 
their true purpose: to support the healthy development 
of individual children and youth (Halpern, 2005). Simi¬ 
larly, an article released by the California Committee on 
Afterschool Accountability argued that afterschool pro¬ 
grams should be valued as “unique institutions” (Piha, 
2006, p. 8) supporting healthy youth development and a 
wide range of learning goals. 

For many funders, results-based accountability has 
fallen short of hopes. Much of the research conducted 
in the late 1990s and early 2000s with the aim of show¬ 
ing the impact of afterschool participation on academic 
achievement, particularly standardized test scores, did 
not meet stakeholder expectations (Bodily & Beckett, 
2005; Dynarski et al., 2004; James- 
Burdumy et al., 2005; Kane, 2004). 
As researchers began to explore 
more deeply the relationship be¬ 
tween program quality and youth 
outcomes (Birmingham, Pechman, 
Russell, & Mielke, 2005; Durlak & 
Weissberg, 2007; Eccles & Goot- 
man, 2002; Grossman, Campbell, 
& Raley, 2007; Miller, 2005; Raley, 
Grossman, & Walker, 2005; Scales 
et al., 2003), their results support¬ 
ed what many in the afterschool 
field already believed: that quality 
is essential to outcomes. 

A New Generation of 
Accountability Systems 

As a result of the growing consen¬ 
sus that program quality is essen¬ 
tial to positive youth outcomes, 
the afterschool field has renewed its focus on how best 
to improve quality (Granger, Durlak, Yohalem, & Reis- 
ner, 2007; Stonehill & Little, 2008). Rather than test¬ 
ing whether programs have produced youth outcomes, 
an increasing number of afterschool funders are shaping 
flexible, collaborative accountability systems designed to 
help programs measure a range of early and intermediate 
outcomes that are better aligned with program goals and 
to strive for higher program quality. 

A key feature of these new lower-stakes accountability 
systems is a subtle but significant shift from viewing pro¬ 
gram improvement as an “add-on” to expecting programs 
to engage in self-assessment and to report on continuous 
improvement efforts. Though producing better youth 
outcomes remains a priority, funders using these new ac- 
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countability systems recognize that program quality and 
efforts to improve it are the essential means to achieving 
this goal. Many public and private funders are now inte¬ 
grating self-assessment into their accountability require¬ 
ments. State-administered 21st CCLC programs provide 
some of the most compelling examples of this shift. 

• In Massachusetts, 21st CCLC grantees are required 
to engage in continuous program improvement using 
the Afterschool Program Assessment System (APAS), 
an integrated set of quality and outcome tools devel¬ 
oped in partnership with the National Institute on 
Out-of-School Time (NIOST). Grantees are required 
to share what they are learning from their APAS data. 
They must report on how they are using their data to 
guide program improvement 
and to increase their capacity 
to produce 21st century skills 
in youth (Massachusetts De¬ 
partment of Elementary and 
Secondary Education, 2011). 

• The Michigan Department of 
Education 21st CCLC program 
uses a “low stakes accountabil¬ 
ity and improvement system” 

(Smith, 2005, p. 5) devel¬ 
oped in collaboration with the 
High/Scope Educational Re¬ 
search Foundation. Programs 
are expected to use the Youth 
Program Quality Assessment for self-assessment 
and to demonstrate that they are using data-driven 
improvement plans and engaging in organizational 
learning. 

• 21st CCLC grantees in Rhode Island are required 
to engage in continuous improvement using a cus¬ 
tomized version of the High/Scope assessment tool 
(Rhode Island Department of Elementary and Sec¬ 
ondary Education, 2006). 

• New York requires 21st CCLC grantees to use the 
New York State Afterschool Network’s Program 
Quality Self-Assessment twice a year for planning 
and ongoing program improvement (New York State 
Education Department, 2011). 

• The Colorado Department of Education uses the 
standardized Monitoring and Quality Improvement 
Tool to evaluate its 21st CCLC grantees. These grant¬ 
ees are required to use this tool once annually as an 
internal self-assessment for planning and quality 
improvement (Colorado Department of Education, 
2011 ). 


Public agencies are joined by large private organi¬ 
zations in a trend toward promoting self-assessment 
and continuous improvement as core components of 
their accountability systems. For instance, the Boys & 
Girls Clubs of America strongly encourages its sites to 
use its Youth Development Outcome Measurement Tool 
Kit (Boys & Girls Clubs of America, 2007). The United 
Way of America, one of the few funders that promoted 
a flexible, collaborative approach to measuring youth 
outcomes from the beginning (United Way of America, 
1996), has for the past two decades supported affiliates 
to use outcome data in order to improve program qual¬ 
ity (Hendricks, Plantz, & Pritchard, 2008). Currently 
NIOST is supporting the use of APAS by United Way af¬ 
filiates in Philadelphia, Boston, and 
Atlanta. These affiliates are using 
components of APAS to help pro¬ 
grams identify appropriate out¬ 
comes, assess quality, and use data 
for continuous improvement. 

Growing evidence suggests 
that engaging in self-assessment can 
indeed lead to higher quality and be 
associated with better outcomes for 
young people. As explained by Weiss 
and Little (2008), self-assessment is 
associated with a “cycle of adapta¬ 
tion” in which afterschool sites col¬ 
lect and analyze data to bring about 
desired quality improvements as part of an ongoing pro¬ 
cess. Two qualitative studies (Pechman & Fiester, 2002; 
Wilson-Ahlstrom, Yohalem, & Pittman, 2007) suggest that 
afterschool staff are likely to view self-assessment findings 
as more credible and useful than results provided by exter¬ 
nal evaluations. Other afterschool studies have found that 
use of quality data motivates change (Akiva & Yohalem, 
2006). Site-level engagement in self-assessment is likely 
to spawn changes in programming (Akiva & Smith, 2007; 
Harris, 2008; Smith, 2005) and, ultimately, improved out¬ 
comes (Sheldon & Hopkins, 2008). 

Though self-assessment can help programs improve, 
simply engaging in self-assessment may not guarantee 
positive results. In fact, a study of self-assessment in 
healthcare names a number of factors necessary to bring 
about the positive effects of self-assessment, such as good 
alignment between self-assessment tools and desired ar¬ 
eas of change, an open and trusting environment between 
frontline staff and supervisors, and constructive feedback 
and support during and following self-assessment (Bose, 
Oliveras, & Edson, 2001). 


Funders that require 
programs to engage in 
self-assessment and to use 
data to improve their 
quality essentially have 
adopted a view of 
afterschool programs not 
simply as deliverers of 
services but as learning 
organizations. 
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Afterschool Programs as Learning 
Organizations 

Funders that require programs to engage in self-assessment 
and to use data to improve their quality essentially have 
adopted a view of afterschool programs not simply as 
deliverers of services but as learning organizations. Peter 
Senge (1990) describes learning organizations as dynamic 
institutions that expand their capacity to achieve results 
by engaging managers and employees in a process that 
helps them strive for personal mastery create mental 
models, adopt a shared vision, promote team learning, 
and practice systems thinking. 

A more recent interpretation of Senge’s work (Garvin, 
Edmondson, & Gino, 2008) suggests that organizations 
can be most effective if they create a supportive learning 
environment where employees feel a sense of psycho¬ 
logical safety, are encouraged to appreciate differences 
and new ideas, and have time for reflection. In success¬ 
ful learning organizations, “leaders actively question and 
listen to employees—and thereby prompt dialogue and 
debate” (Garvin, Edmondson, & Gino, 2008, p. 113). 
Organizational change must happen at the ground level, 
not just at the top. Moynihan (2005) suggests that orga¬ 
nizations are more likely to learn from their data when 
“routines of data collection and dissemination are fol¬ 
lowed by routines of information use” (p. 203) through 
learning forums. 

A look at research findings from studies in education 
(Eccles & Roeser, 1999; Pianta, 2003; Pianta & Hamre, 
2009; Pressley et al., 2003), youth development (Eccles 
& Gootman, 2002), organizational learning (Garvin, Ed¬ 
monson, & Gino, 2008; Senge, 1990), and practitioner 
self-assessment (Bose, Oliveras, & Edson, 2001; Wilson- 
Ahlstrom, Yohalem, & Pittman, 2007) suggest that many 
of the practices recommended for supporting learners cut 
across disciplines. While there are variations in specific 
beliefs and approaches, three main domains of effective 
learning environments appear to be common across dis¬ 
ciplines: supportive social environments, opportunities 
for skill building, and appropriate structure and expecta¬ 
tions (Figure 1). 

Accountability Systems Designed to Support 
Afterschool Programs as Learning Organizations 

The new generation of accountability systems is bridging 
the arenas of continuous quality improvement and ac¬ 
countability. By acknowledging the central importance of 
quality, aligning outcomes with program practices, and 
using self-assessment as a driver for change, the funders 
and sponsors using these approaches to accountability 


Figure 1. Elements of Effective Learning Environments 



are likely to produce better results than will funders that 
require afterschool programs to report on academic and 
other long-term outcomes for youth. 

For decades, the field of afterschool has sought to 
support childrens positive development by creating 
quality standards for developmentally appropriate envi¬ 
ronments and by executing research-supported practices 
for advancing childrens learning. These same principles 
can support the development of afterschool programs as 
learning organizations striving to improve quality. 


Figure 2. Cycle of Afterschool Organizational Learning 
and Improvement 
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Supporting a Cycle of Afterschool 
Program Learning 

Many funders, sponsors, and intermediary organiza¬ 
tions, recognizing that programs need to learn, provide 
support, resources, and training to help programs use 
self-assessment. However, if architects of accountability 
systems are serious about improving program quality 
with an eye toward producing better youth outcomes, 
they should explicitly embrace the notion of afterschool 
programs as learning organizations. 

Funders and sponsors that want to support pro¬ 
grams as learning organizations could begin by recogniz¬ 
ing the steps of the cycle of organizational learning and 
continuous program improvement (Figure 2). Many af¬ 
terschool researchers and groups suggest a similar cycle, 
whose key steps are goal setting, planning, data collec¬ 
tion, analysis, reflection, improvement, and reassessment 
(NIOST, 2011; Sheldon & Hopkins, 2008; Surr, Behler, 
& Milla-Lugo, 2009; Weiss & Little, 2008). 


Figure 4. Accountability System Element: Supportive 
Social Environment 



Accountability System Elements to 
Support the Learning Cycle 

Drawing on research in organization development and 
education, funders and sponsors, in partnership with in¬ 
termediary organizations, could explicitly support each 
step in this cycle by providing the key elements associat¬ 
ed with effective learning environments (Figure 3). After¬ 
school programs receiving this combination of supports 
are more likely to become fully engaged “learners” and 
therefore to execute the quality improvements needed to 
produce positive youth outcomes. 

Figure 3. Accountability System Elements to 
Promote Afterschool Learning and Improvement 


Supportive Social Environment 

Research suggests that learning is more likely to take place 
in a supportive social environment (Akey, 2006; Benard, 
1996; Bransford, Brown, & Cocking, 2000; Connell & 
Gambone, 2002; Eccles & Gootman, 2002; Pianta, 2003). 
Afterschool funders and sponsors have traditionally re¬ 
lied on intermediary organizations to support programs 
by providing training, coaching, and technical assistance. 
This approach has many advantages: programs get sup¬ 
port from individuals with true expertise in the field while 
feeling free to acknowledge areas of weakness without 
fearing loss of funding. Figure 4 suggests how funders and 
sponsors can retain their monitoring role while fostering a 
supportive social environment for programs. 
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Safe psychological environment. A lower-stakes fund¬ 
ing environment, in which funders do not penalize pro¬ 
grams for reporting less than desirable results and refrain 
from comparing programs publicly, may help programs 
to engage more authentically in self-assessment. 

Encouragement, feedback, and problem-solving help. 

Program leaders need support, constructive feedback, 
and help with improvement priorities from an external, 
seasoned expert. Traditionally this supportive person has 
been a coach, trainer, or technical assistance provider 
from an intermediary organization. Funding that enables 
these external supports to continue will benefit program 
learning. 

Opportunities for peer support and positive social 
norms. Many state and city initiatives provide peer net¬ 
working opportunities. Funders, sponsors, and interme¬ 
diaries can help to create positive social norms for as¬ 
sessment by, for example, publicizing examples of how 
programs are integrating continuous improvement into 
their practice. 

Opportunities for Skill Building 

Program assessment and improvement are not innate 
skills. In fact, the skills needed to self-assess, collect data, 
and interpret and use that data represent a completely 
different skill set from the curricular, instructional, and 
administrative competencies afterschool professionals 
are expected to have. By providing opportunities for 
program leaders to master these skills and requiring that 
they demonstrate how they are incorporating assessment 
into their everyday practice, funders and sponsors can 
increase the likelihood that programs become learning 
organizations, achieve higher levels of quality, and ulti¬ 
mately produce better youth outcomes. Figure 5 and its 
description below suggest how funders, sponsors, and 
intermediary organizations can promote program skill 
building. 

Build assessment literacy. Many (probably most) pro¬ 
gram administrators need to build skills in identifying ap¬ 
propriate outcomes, selecting measurement tools, using 
data management systems, and analyzing and interpret¬ 
ing data. While many funders and intermediaries provide 
one-day workshops and general support for these activi¬ 
ties, program leaders need more explicit, intensive, and ex¬ 
tended instruction to master these tasks (Lukin, Bandolos, 
Eckhout, & Mickelson, 2004). One recent initiative, the 
Boston Capacity Institute, works with youth-serving orga¬ 


Figure 5. Accountability System Element: 
Opportunities for Skill Building 
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nizations to assess and strengthen their data collection and 
performance management systems through a rigorous two- 
year support process (Black Ministerial Alliance of Greater 
Boston, 2011). Such intensive and focused interventions 
are comparatively rare. When providing programs with as¬ 
sessment literacy internally is not feasible, funders could 
encourage cohorts of funded programs to pursue “insourc¬ 
ing," in which programs share an external evaluator while 
practitioners focus on learning to understand and use ac¬ 
tionable data (Miller, Kobayashi, & Noble, 2006). 

Elelp programs integrate self-assessment into ongoing 
practice. One of the more promising methods of self- 
assessment is observation followed by structured reflec¬ 
tion and a discussion of practice (Seidman, Tseng, & 
Weisner, 2006; Smith, 2005; Surr, Behler, & Milla-Lugo, 
2009). Yet few program directors know how to conduct 
an observation, and fewer know how to lead staff in re¬ 
flection on and discussion of practice. Another critical 
skill is the ability to articulate clear goals and devise real¬ 
istic action plans on an ongoing basis (Moynihan, 2005). 
Funders and sponsors can help by asking leaders to com¬ 
municate their improvement goals, to create written ac¬ 
tion plans that are clearly linked to their data findings, 
and to articulate how they are incorporating assessment 
into everyday practice. 

Elelp directors learn to lead improvement efforts and 
engage staff in the assessment process. To facilitate 
program improvement, change must take place on the 
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front lines where staff members deliver programming to 
youth. Funders and intermediary organizations should 
set the expectation that assessment efforts will engage 
frontline staff and should teach administrators to train 
their staff in self-assessment. 

Provide professional development that uses research- 
based instructional practices. Too many professional 
development workshops rely on written materials, slide 
presentations, and lecture rather than using the instruc¬ 
tional methods research says will engage practitioners. 
Adult learners, like children and youth, respond best to 
teaching practices that are developmentally appropriate, 
that engage them in interactive and cooperative learning, 
and that help them construct meaning and build under¬ 
standing from their existing knowledge and skill base. 

Appropriate Structure and Expectations 

Research suggests that effective learning environments 
balance a high degree of structure, rules, and routines 
with opportunities for learners to contribute, make deci¬ 
sions, and exercise their autonomy (Eccles & Gootman, 
2002). Funders and sponsors are in a unique position 
to offer programs the right blend of structure and flex¬ 
ibility and to set expectations that will lead to success. 
The building blocks of appropriate structure and expec¬ 
tations are outlined in Figure 6 and below. 

Offer voice, choice, and opportunities for contribu¬ 
tion and decision making. Like children and youth, 
afterschool practitioners need voice and choice in order 
to engage fully in learning. Giving them at least some 
choice in selecting areas for improvement and allowing 
them to target short-term outcomes appropriate to their 
programs will increase buy-in so that the data collected 
will reflect program goals and actually be used to inform 
practice. When appropriate, offering program adminis¬ 
trators and staff the chance to contribute to or give feed¬ 
back on accountability expectations will enhance their 
motivation to meet those expectations. 

Set reasonable expectations for data collection. When 
they collect too much data, administrators and staff have 
difficulty understanding and using the information (Fi- 
ester, 2004; Flarris, 2008; Sternberg, 2006). If practi¬ 
tioners are involved in decisions about which and how 
much data to collect—if they are encouraged to limit the 
amount of data they collect and to articulate the research 
questions the data will help them answer—then they will 
be more likely to use assessment data for change. 


Figure 6. Accountability System Element: Appropriate 
Structure and Expectations 
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Offer enough time on task. Program improvement 
must be recognized as an integral piece of quality youth 
programming. Funders must, therefore, provide money 
to allow programs to dedicate paid staff time to assess¬ 
ment and improvement. Otherwise, these activities will 
continue to take a back seat to program operations and 
other daily responsibilities. Ideally large programs would 
dedicate one experienced staff person to lead assessment 
and improvement activities. 

Set high, achievable, and developmentally appro¬ 
priate expectations. As we know from the fields of 
education and youth development, one key element 
for effective learning is communicating, and holding 
learners to, high expectations (Benard, 1996; Eccles 
& Gootman, 2002). Learners respond best to expec¬ 
tations and learning goals that fit their developmental 
levels, are appropriately challenging, and can realisti¬ 
cally be achieved (Akey, 2006; Bransford, Brown, & 
Cocking, 1999). Most afterschool programs will experi¬ 
ence more success if, rather than striving for better stu¬ 
dent test scores, they promote appropriate short-term 
outcomes—such as youth engagement, social skills, 
and problem solving—that are linked to longer-term 
academic outcomes. Funders that set clear, high, and 
appropriate expectations for programs can motivate ad¬ 
ministrators and staff to keep improvement efforts on 
the front burner. 
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A New Direction for Accountability Systems 

In the end, programs have to be held accountable for 
how they are benefiting the youth they serve. Given the 
overwhelming evidence that high-quality programs are 
essential to helping our children learn, funders and spon¬ 
sors should uphold high expectations for quality, and, 
ultimately, for appropriate and realistic youth outcomes. 
But high expectations and accountability for outcomes 
alone are not enough. Accountability systems that em¬ 
brace afterschool programs as learning organizations and 
offer them the structure, skill-building opportunities, 
and support they need to improve quality are most likely 
to succeed in their goal of achieving better outcomes for 
young people. 
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